feat: Add refresh to get get updated TableMetadata#1154
feat: Add refresh to get get updated TableMetadata#1154jonathanc-n wants to merge 5 commits intoapache:mainfrom
refresh to get get updated TableMetadata#1154Conversation
|
Is this on track with what we want? cc @liurenjie1024 @Fokko @ZENOTME |
|
@jonathanc-n This is the very first step, and we indeed want this. I don't think this is the actual conflict detection. When between the |
ZENOTME
left a comment
ZENOTME
left a comment
There was a problem hiding this comment.
Thanks @jonathanc-n! I think this PR can be more about add refresh op for table. And as I describe below, it may more appropriate to delay the transaction part after #949.
|
|
||
| /// Creates a fast append action. | ||
| pub fn fast_append( | ||
| pub async fn fast_append( |
There was a problem hiding this comment.
According to iceberg-java, looks like we don't need to refresh here. The lifecycle of a transaction looks like this:
- create transaction and get current table
- apply action to current table
- apply action to current table
- commit transaction
- refresh and get the latest table, see: https://github.com/apache/iceberg/blob/c661a71091e496393c743ddd879d9e1a0f2747b2/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L503
- reapply all update before and commit to catalog
I think can delay this part after #949. cc @Fokko @liurenjie1024
jonathanc-n
changed the title
refresh to get get update TableMetadata
jonathanc-n
changed the title
refresh to get get update TableMetadatarefresh to get get updated TableMetadata
|
Is there an easy way to be able to create dummy catalog instances and pass them into new table instances for testing? |
4 participants
Which issue does this PR close?
What changes are included in this PR?
Added basic conflict detection. The current idea is that before commit we can use catalog to refresh the table state before commit to compare with the base metadata.